D-Expressions: Lisp Power, Dylan Style

syntax trees (AST’s) are highly structured representations of source code. In an object-oriented implementation, there would typically be one class per definition, statement, or other language construct. Down at the expression level, there might be classes corresponding to function calls, array accesses, literal references, and so on. Typically with an AST, adding a new syntactic construct requires extending the parser to understand it and adding a new class to represent that construct. How difficult this is is related to the complexity of the parser because there is no intermediate form between tokens and full AST. Worst case, there is no special support at all for modular macro extensions, and new language constructs must be defined as a cooperating element of the full language grammar [3]. It could require extensive knowledge of the parser’s intricacies to avoid or resolve problems like shift/reduce conflicts in an LALR parser, for example. We don’t consider requiring such knowledge of programmers to be reasonable. 3.4 Skeleton Syntax Tree Representations The distinguishing feature of a skeleton syntax tree (SST) is that it has few classes and corresponds to a very simple core grammar. That grammar describes “shapes” rather than particular language constructs. Many different constructs may fit within the same basic shape. The most extreme example of a language based on an SST is, of course, Lisp. Lisp’s underlying grammar says nothing about cond or defun, it simply specifies the single basic shape of all Lisp constructs. Although not a programming language, XML has much the same property. SST’s define the axioms to which all syntax must conform. In Lisp, it is, essentially, that brackets match. In XML, that tags must match. The simplicity of the axioms makes the overall structure of the language easy for people to understand and to remember and also for programs to work with. Language-level constructs are defined modularly, above the core grammar and without requiring any changes to it. User-defined constructs have the same status as standard syntactic forms. Referring back to a previous section, another way to look at an SST is as a lightly-structured, token-based representation. SST’s have most of the power of general token-based approaches, but without many of the pitfalls discussed. The challenge is to define a simple SST upon which it is possible to build a language with a conventional algebraic syntax. It is worth noting that although Lisp is SST-based, the SST used in most Lisps would not meet our requirements. Because standard run-time objects (numbers, symbols, lists) are reused to represent the Lisp SST, there is no reliable way to record auxiliary information like source location or hygiene context. Scheme macro systems [5] that implement automatic hygiene have to face this deficiency and work in terms of special-purpose syntax objects rather than lists and symbols. 4 Dylan’s Rewrite-Rule Only Macro System Dylan macros provide a way to define new syntactic forms. Their definitions are called macro definitions and their usages are called macro calls. At compile-time, macro expansion replaces macro calls with other constructs per their definition. This macro expansion process is repeated until the resulting expansion contains no more macro calls. 4.1 Dylan’s Rewrite Rules The macro system defined in the Dylan Reference Manual (DRM) [13] is a rewrite-rule only pattern matching template substitution system. Macros are named with module bindings using the usual Dylan namespace mechanism. Each macro is comprised of a sequence of rewrite rules where the left-hand side of a rule is a pattern that matches a fragment of code and the right-hand side of a rule is a template which is substituted for the matched fragment. Pattern variables in the left-hand side serve as macro arguments. Pattern variables appearing on the right-hand side substitute arguments into the expansion. The rules are attempted in order until a given rule’s pattern is found to match, at which time, the matched fragment is replaced with the corresponding template. If no patterns are found to match, then an error is raised. 4.2 Dylan’s Skeleton Syntax Tree Before we delve into the intricacies of the rewrite-rule only macro system, it is important to explore Dylan’s SST. Dylan’s SST is a richer version of Lisp’s s-expression based SST. Lisp’s SST can roughly be described as follows: form: symbol form: literal form: "(" elements ")"